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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1 .1 14, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1 .1 14. Applicant's submission filed on 

1 1/28/2005 has been entered. 

Information Disclosure Statement 

2. The 12/05/2005 IDS directed the examiner's attention to copending application 
SN 09/648,463 which has been reviewed by this examiner in response to the IDS on 
2/14/2006. 

Response to Amendment 

3. The amendment filed on 1 1/28/2005 has been entered. The amendments to the 
claims overcomes the previous 35 USC 112 second paragraph rejection. 

Response to Argument 

4. Applicants arguments filed on 1 1/28/2005 have been fully considered, but they 
are not deemed to be persuasive because Kitamura article teaches in section 2 using 
range image or stereo image to bring a real image of an object that exists in the real 
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world into a computer generated virtual world which is also discussed by applicant in 
paragraph 0028. Paragraph 0028 states: 

[0028] A constraining shape input unit 1060 obtains the three-dimensional position of a 
constraining plane 2000 in real space, and inputs this to a constraining shape 
generating unit 1080. The present embodiment uses the stylus of the 3D SPACE 
FASTRAK sensor manufactured by Polhemus. This stylus is provided with one button 
(not shown) serving as a user interface for specifying the three-dimensional position of 
objects in the real space, and so forth. The present invention is not restricted to such 
three-dimensional pointing devices, rather, any arrangement may be applied as long as 
constraining shapes in real space can be precisely modeled, such as a method wherein 
image characteristics of real space are taken with a stereo camera to obtain the shape 
of an object. 

Thus, Kitamura teaches "obtaining a constraining shape from a plurality of positions in 
real space (The measurements by the user in three dimensional real space, the device 
for obtaining the 3-D shape by using a range image, and the device for obtaining the 3- 
D shape by using multiple cameras.) designated by a user (The user measures, the 
user directs the range measuring device to locations in the real world, and the user 
directs the cameras to locations in the real world) using an operating unit (Ruler or 
range device or multiple camera device.) capable of obtaining three-dimensional 
positional information (Each of the devices, ruler, range and multiple camera obtain 
three-dimensional information.)". 

The indication of allowability of claims 1 -9 and 1 6-1 9 is withdrawn in view of a 
better understanding of the claims, disclosure, the Kitamura article and the broadening 
effect applicants paragraph 0028 has on the claimed invention. 
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Claim Rejections - 35 USC § 102 

5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

6. Claims 1-3 and 5-18 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Yoshifumi Kitamura and Fumio Kishino, Consolidated Manipulation of Virtual and Real 
Objects, September 1997, Proceedings of the ACM symposium on Virtual reality 
software and technology, pages 133-138. Kitamura teaches an augmented reality 
system that uses object constraints to control the visual interaction between the virtual 
objects and the real objects. Claim 10 does not have an image taking step and thus 
does not claim the obtaining a constraining shape is separate from the image taking 
step. Therefore Kitamura teaches this broad method. 

A detailed analysis of the claims follows. 

Claim 10: 

Kitamura teaches an information processing method for changing the position 
and orientation of a virtual object in mixed reality space obtained by combining a real 
image and a virtual image, said method comprising the steps of: 

obtaining a constraining shape from a plurality of positions in real space {The 
measurements by the user in three dimensional real space, the device for obtaining the 
3-D shape by using a range image, and the device for obtaining the 3-D shape by using 
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multiple cameras.) designated by a user (The user measures, the user directs the range 
measuring device to locations in the real world, and the user directs the cameras to 
locations in the real world.) using an operating unit (Ruler or range device or multiple 
camera device.) capable of obtaining three-dimensional positional information (Each of 
the devices, ruler, range and multiple camera obtain three-dimensional information.)" . 
(The three paragraphs found in section 2 on page 134 ofKitamura teaches the user 
measuring the real objects, a device measuring the real objects with range in response 
to the user, or a device measuring the real objects with multiple cameras range in 
response to the user. Kitamura discusses in section 2 in the first paragraph using 
conventional modeling software after precisely measuring the size or length of the real 
object by hand or by the devices which is an operating unit capable of obtaining three- 
dimensional positional information . ); 

changing the position and orientation of the virtual object according to 
instructions from the user, based on the obtained constraining shape as constraint 
conditions (The first paragraph in section 5 on page 135 describes the user using a 6 
DOF tracker device to manipulate the virtual objects. The shape of the real objects are 
used to constrain the movement of the virtual object by giving the real object a shape 
that the virtual object interacts with in a constrained manner. Sections 5. 1 to 5.4 
discusses manipulation of the virtual object based upon constraint conditions based on 
the shape of the real object. The shape generated from the external instructions 
constraints the interaction of the virtual object with the. real object, see sections 5. 1 to 
5.4.); and 
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combining an image of the virtual object generated according to the changed 
position and orientation, and the real image (The introduction on page 1 33 second full 
paragraph discusses augmented reality which synthesizes a virtual object with a real object. 
The claim does not limit the form of the real image, thus, Kitamura teaches this limitation. 
Section 5. 1 discusses after movement is detected by the 6DOF manipulator the virtual object 
is moved according to the constraints.). 

Claim 11: 

Kitamura teaches an information processing method according to Claim 10, 
further comprising the step of combining a virtual image indicating the constraining 
shape with the real image (The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a real object. The claim does not 
limit the form of the real image, thus, Kitamura teaches this limitation. Section 5. 1 discusses 
after movement is detected by the 6DOF manipulator the virtual object is moved according to 
the constraints.). 

Claim 12: 

Kitamura teaches an information processing method according to Claim 10, 
wherein the constraining shape is a plane. (Onpage 136 in the text above figure 2 determining a 
plane and using the plane to constrain movement of the virtual object is discussed with regards to figure 
2). 
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Claim 13: 

Kitamura teaches an information processing method according to Claim 10, 
wherein said changing step changing the position and orientation of the virtual object is 
carried out by changing the position and orientation of the operating unit (The 6 DOF 
tracker is an operating unit. The user using the 6 DOF tracker device manipulates the 
virtual objects.). 

Claim 14: 

Kitamura teaches a computer program product comprising a computer readable 
medium storing computer program code for performing the information processing 
method according to Claim 10 wherein the information processing method is executed 
by a computer device (This article is directed to computers that generate the augmented 
reality scene since it was published by ACM for a symposium on virtual reality software 
and technology and since at page 133 in the last sentence in the second paragraph of 
section 1 software/hardware is discussed. Software causing a computer to perform 
Kitamura's augmented reality is inherently stored in a computer readable recording 
medium.). 

Claim 15: 

Kitamura teaches a computer-readable recording medium, storing the computer 
program according to Claim 14 (Software causing a computer to perform Kitamura's 
augmented reality is inherently stored in a computer readable recording medium.). 
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Claim 16: 

The functions of Kitamura corresponds to the claimed units because the software 
causes the computer to become a unit that performs a process. As seen below 
Kitamura performs the claimed processes. 

Kitamura teaches an information processing device for aiding control operations 
relating to controlling the position and orientation of a virtual object, said device 
comprising: 

an image capturing unit configured to capture a real image in real space (The 
measurements by the user in three dimensional real space, the device for obtaining the 
3-D shape by using a range image, and the device for obtaining the 3-D shape by using 
multiple cameras captures a real image in real space. The three paragraphs found in 
section 2 on page 134 of Kitamura teaches the user measuring the real objects, a device 
measuring the real objects with range in response to the user, or a device measuring the 
real objects with multiple cameras range in response to the user. The claim does not limit 
the form of the real image, thus, Kitamura teaches this limitation.)] 

a virtual image generation unit for generating a virtual image of a virtual object 
according to the position and orientation of said image capturing unit (Kitamura 
discusses in section 2 in the first paragraph using conventional modeling software after 
precisely measuring the size or length of the real object by hand or by the devices which 
is an generation unit capable of generating three-dimensional positional information . ) ; 

a superimposing unit configured to superimpose the generated virtual image with 
the captured real image (Section 2 discusses mixing the virtual and real world objects.); 
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an inputting unit configured to input three-dimensional position information of a 
plurality of positions inputted by a user in the real space (The measurements by the 
user in three dimensional real space, the device for obtaining the 3-D shape by using a 
range image, and the device for obtaining the 3-D shape by using multiple cameras are 
used to configure the constraints on the real world objects by using the three 
dimensional coordinates of the real world shape. ); 

a setting unit configured to set a constraining shape based on the inputting three- 
dimensional position information (The first paragraph in section 5 on page 135 describes 
the user using a 6 DOF tracker device to manipulate the virtual objects. The shape of the 
real objects are used to constrain the movement of the virtual object by giving the real 
object a shape that the virtual object interacts with in a constrained manner.) ; and 

an operating unit configured to control the position and orientation of the virtual 
object based on the constraining shape in accordance with the a user's instruction^ 
Sections 5. 1 to 5.4 discusses manipulation of the virtual object fiased upon constraint 
conditions based on the shape of the real object. The shape generated from the external 
instructions constrains the interaction of the virtual object with the real object see 
sections 5. 1 to 5.4. The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a real object. Section 5. 1 discusses 
after movement is detected by the 6DOF manipulator the virtual object is moved according to 
the constraints.) 
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Claim 17: 

Kitamura teaches an information processing device according to Claim 16, 
wherein the constraining shape is defined by polygons (See section 4.1 second 
paragraph. Polyhedria includes polygons.) and the apexes of the polygons are at 

positions inputted by the user (According to section 2 the input points are used to form 

r 

the real objects in the virtual world, thus, the apexes of the polyhedria correspond to the 
position inputted by the user by use of the ruler, range device, or device using multiple 
cameras.) or the constraining shape is a plane passing through the positions inputted by 
the user (A real sensed surface is a planar real world object). 

Claim 18: 

Kitamura teaches an information processing device according to Claim 16, 
wherein said operating unit performs at least one of the following operations in 
performing an operation controlling the position and orientation of the virtual object: 

a translation operation for causing translational movement of the virtual object 
based on the constraining shape (Page 137 column 1 lines 1-3 teaches translation 
when the virtual object collides with the real surface.)] or 

a rotation operation for rotating the virtual object on an axis which is a normal 
vector at a plane where the constraining shape and the virtual object come into contact 
(Page 137 column 1 lines 1-3 teaches rotation when the virtual object collides with the 
real surface.) (Page 137 column 1 lines 1-3 also teaches translation and rotation when 
the virtual object collides with the real surface). 
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Claims 1-3: 

Means plus function claims 1-3 correspond to device claims 16-18 and the 
means of Kitamura, software and computer, are equivalent to applicant's means of 
software and computer. 

Claims 5-7: 

Step for claims 5-7 correspond to device claims 16-18 and the steps of Kitamura, 
software and computer, are equivalent to applicant's steps performed by software and 
computer. 

Claims 8 and 9: 

Claims 8 and 9 mirror claims 14 and 15 addressed above and they are rejected 
for the same reasons given above for claims 14 and 15. 



Prior Art 

7. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 



The article by Yoshifumi Kitamura, Amy Yee, and Fumio Kishino, titled A 
Sophisticated Manipulation Aid in a Virtual Environment using Dynamic Constraints 
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among Object Faces Presence, Vol. 7, No. 5, October 1998, by the Massachusetts 
Institute of Technology pages 460-477 . 

The article by Gudrun Klinker, Didier Strieker, and Dirk Reiners titled Augmented 
Reality: A Balance Act between High Quality and Real-Time Constraints, 1999 by 
Technical University of Munich, Germany, Fraunhofer Project Group for Augmented 
Reality at ZGDV, Germany. 

The article by Axel Pinz titled Consistent Visual Information Processing Applied 
to Object Recognition, Landmark Definition, and Real-Time Tracking, 2001, by Graz 
University of Technology, Institute of Electrical Measurement and Measurement Signal 
Processing, Schieftstattgasse 14B, A-8010 Graz, Austria. 

8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is (571 ) 272- 
7656. The examiner can normally be reached on M-F from 7:00 to 3:30. If attempts to 
reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
Razavi, can be reached at (571 ) 272-7664. The fax phone Number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
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Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-21 7-91 97 (toll-free). 
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Primary Examiner 
Art Unit 2672 



